{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a8231e26-6759-406d-aa95-17e5f48d68a2",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "\n",
    "In order to use Llama2, you need access from Meta. We suggest following the steps in this [article](https://medium.com/@yashambekar1804/getting-started-with-llma2-accessing-llama2-70b-model-and-obtaining-hugging-face-api-token-and-8773e19bda26). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9fb5d47-d6b9-4cdd-b2b4-80dda4eef996",
   "metadata": {},
   "source": [
    "# HR Chatbot using Retrieval Augmented Generation\n",
    "\n",
    "Many companies aim to make HR documents more accessible to employees and contractors. A recent promising approach is building a RAG system on top of HR documents. This is because it\n",
    "- provides a unique response to any query,\n",
    "- reduces hallucinations as the answer has to be grounded in the retrieved context,\n",
    "- makes the process highly scalable due to automatic question-answering.\n",
    "\n",
    "However, this solution possesses its unique challenges. For example\n",
    "- keeping the knowledgebase consistent and up-to-date,\n",
    "- running LLMs efficiently,\n",
    "- ensuring the generated results are correct and aligned with company communication guidelines,\n",
    "\n",
    "In our case, we have three HR policy sources for our hypothetical company.\n",
    "1. An older HR policy, which contains our maternity leave policy and details on manager responsibilities.\n",
    "1. A more recent update, that contains inaccurate information on management responsibilities - but one that also contains unique information on paternity leave.\n",
    "1. A newer policy document that contains updated information about management responsibilities, correcting the mistakes of the previous modernizing attempt - alongside some other HR policy information.\n",
    "\n",
    "A good system will be able to:\n",
    "- provide relevant information on maternity leave (only covered in the old document),\n",
    "- synthesize contradicting information and only present to us the correct ones\n",
    "\n",
    "Regarding synthesizing information, there are contradictions between the documents pertaining to management responsibilities. We are going to use the \n",
    "- creation_date of the policy,\n",
    "- and the usefulness score of each paragraph\n",
    "\n",
    "as a proxy to know for similar documents with \n",
    "\n",
    "- similar information but different wording, and some important parts slightly altered, or\n",
    "- also talking about seemingly the same topic, but without conveying useful information\n",
    "\n",
    "which one is more relevant to our query."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e296da7d-1bce-4bd4-8edf-ff0ce00addde",
   "metadata": {},
   "source": [
    "## Boilerplate\n",
    "\n",
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0176e0bf-0e5c-46cb-8483-401cb7055039",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%pip install superlinked==37.5.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033c537c-51df-4518-bdf9-851d6c95fec0",
   "metadata": {},
   "source": [
    "### Imports and constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "695ab4be-2a2b-4ea0-bae4-8de17f610ea3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime, timedelta\n",
    "import os\n",
    "\n",
    "import altair as alt\n",
    "import pandas as pd\n",
    "from torch import float16\n",
    "from transformers import AutoTokenizer, pipeline\n",
    "from superlinked import framework as sl\n",
    "\n",
    "alt.renderers.enable(sl.get_altair_renderer())\n",
    "alt.data_transformers.disable_max_rows()\n",
    "pd.set_option(\"display.max_colwidth\", 1000)\n",
    "START_OF_2024_TS = int(datetime(2024, 1, 2).timestamp())\n",
    "EXECUTOR_DATA = {sl.CONTEXT_COMMON: {sl.CONTEXT_COMMON_NOW: START_OF_2024_TS}}\n",
    "TOP_N = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9493a0-77b7-494b-8fb5-3b9991237404",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3adad65b-c9e2-40f8-aaec-fb157267988d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>body</th>\n",
       "      <th>creation_date</th>\n",
       "      <th>usefulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>1672527600</td>\n",
       "      <td>0.555771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>1672527600</td>\n",
       "      <td>0.310004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>1672527600</td>\n",
       "      <td>0.410012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Each quarter, all employees of &lt;Company Name&gt; will undergo a performance evaluation encompassing an appraisal of job performance, competency within their respective roles, adherence to company regulations and ethical standards, and attainment of role-specific goals.</td>\n",
       "      <td>1672527600</td>\n",
       "      <td>0.389284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>To sustain a high level of security awareness, the organization will disseminate security updates and communications to its personnel occasionally via various communication platforms as necessary.</td>\n",
       "      <td>1672527600</td>\n",
       "      <td>0.594588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  \\\n",
       "0      0   \n",
       "1      1   \n",
       "2      2   \n",
       "3      3   \n",
       "4      4   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                       body  \\\n",
       "0          Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "1                                                                                                                                                                                                                                     Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "2  Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "3                                                                                                                Each quarter, all employees of <Company Name> will undergo a performance evaluation encompassing an appraisal of job performance, competency within their respective roles, adherence to company regulations and ethical standards, and attainment of role-specific goals.   \n",
       "4                                                                                                                                                                                      To sustain a high level of security awareness, the organization will disseminate security updates and communications to its personnel occasionally via various communication platforms as necessary.   \n",
       "\n",
       "   creation_date  usefulness  \n",
       "0     1672527600    0.555771  \n",
       "1     1672527600    0.310004  \n",
       "2     1672527600    0.410012  \n",
       "3     1672527600    0.389284  \n",
       "4     1672527600    0.594588  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_df = pd.read_csv(\"https://storage.googleapis.com/superlinked-notebook-hr-knowledgebase/hr_rag_policies.csv\")\n",
    "text_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06713661-a095-4159-82cf-acf9763c313f",
   "metadata": {},
   "source": [
    "Our dataset contains documents from all 3 sources, with the following columns:\n",
    "- index,\n",
    "- body: text of the HR policy paragraph,\n",
    "- creation_date: utc timestamp when the policy was issued,\n",
    "- usefulness: the ratio of people deeming the paragraph useful when prompted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5324f0d-a3ae-483a-a9a4-97c784bcd324",
   "metadata": {},
   "source": [
    "<style>\n",
    ".table_component {\n",
    "    overflow: auto;\n",
    "    width: 100%;\n",
    "}\n",
    "\n",
    ".table_component table {\n",
    "    border: 1px solid #dededf;\n",
    "    height: 100%;\n",
    "    width: 100%;\n",
    "    table-layout: fixed;\n",
    "    border-collapse: collapse;\n",
    "    border-spacing: 1px;\n",
    "    text-align: left;\n",
    "}\n",
    "\n",
    ".table_component caption {\n",
    "    caption-side: top;\n",
    "    text-align: left;\n",
    "}\n",
    "\n",
    ".table_component th {\n",
    "    border: 1px solid #dededf;\n",
    "    background-color: #eceff1;\n",
    "    color: #000000;\n",
    "    padding: 5px;\n",
    "}\n",
    "\n",
    ".table_component td {\n",
    "    border: 1px solid #dededf;\n",
    "    background-color: #ffffff;\n",
    "    color: #000000;\n",
    "    padding: 5px;\n",
    "}\n",
    "</style>\n",
    "<div class=\"table_component\" role=\"region\" tabindex=\"0\">\n",
    "<table>\n",
    "    <caption>\n",
    "        <p><b>Breakdown of policy documents</b></p>\n",
    "    </caption>\n",
    "    <thead>\n",
    "        <tr>\n",
    "            <th><b>Document</b></th>\n",
    "            <th><b>Relevance</b></th>\n",
    "            <th><b>Recency</b></th>\n",
    "            <th><b>Usefulness</b></th>\n",
    "        </tr>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr>\n",
    "            <td>Old document<br></td>\n",
    "            <td>Management responsibilities, maternity leave<br></td>\n",
    "            <td>Low<br></td>\n",
    "            <td>Mid-high<br></td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>Recent, but low quality</td>\n",
    "            <td>Management responsibilities, paternal leave</td>\n",
    "            <td>High</td>\n",
    "            <td>Low, but high for paternity leave</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "            <td>Recent, fixed</td>\n",
    "            <td>Management responsibilities</td>\n",
    "            <td>High</td>\n",
    "            <td>High</td>\n",
    "        </tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f531cd-2a14-4cc4-a045-2875d4246d80",
   "metadata": {},
   "source": [
    "## Set up Superlinked for Retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "15c6b58a-9b46-4a72-97bc-94b3fa1c3c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# typed schema to describe our inputs\n",
    "class ParagraphSchema(sl.Schema):\n",
    "    body: sl.String\n",
    "    created_at: sl.Timestamp\n",
    "    usefulness: sl.Float\n",
    "    id: sl.IdField"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "68df0aca-c63a-4b30-af2f-ad40313c39f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraph = ParagraphSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "09e0fee5-9206-4a92-95df-aacbbed32f96",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# relevance space will encode our knowledgebase utilizing chunking to control the granularity of information\n",
    "relevance_space = sl.TextSimilaritySpace(\n",
    "    text=sl.chunk(paragraph.body, chunk_size=100, chunk_overlap=20),\n",
    "    model=\"sentence-transformers/all-mpnet-base-v2\",\n",
    ")\n",
    "# recency has a periodtime to differentiate between the document created at the beginning of this year and last year\n",
    "recency_space = sl.RecencySpace(\n",
    "    timestamp=paragraph.created_at,\n",
    "    period_time_list=[sl.PeriodTime(timedelta(days=300))],\n",
    "    negative_filter=-0.25,\n",
    ")\n",
    "# usefulness is a ratio, so the relevant range is [0-1], and we prefer higher numbers\n",
    "usefulness_space = sl.NumberSpace(number=paragraph.usefulness, min_value=0.0, max_value=1.0, mode=sl.Mode.MAXIMUM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "38b58202-7f4a-4dde-82df-0bacee012945",
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraph_index = sl.Index([relevance_space, recency_space, usefulness_space])\n",
    "paragraph_parser = sl.DataFrameParser(paragraph, mapping={paragraph.id: \"index\", paragraph.created_at: \"creation_date\"})\n",
    "source: sl.InMemorySource = sl.InMemorySource(paragraph, parser=paragraph_parser)\n",
    "executor = sl.InMemoryExecutor(sources=[source], indices=[paragraph_index], context_data=EXECUTOR_DATA)\n",
    "app = executor.run()\n",
    "source.put([text_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bfe88caf-e5ec-4ce7-8665-501b95fa8b02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# our simple query will make a search term possible, and gives us the opportunity to weight input aspects\n",
    "# relevance, recency and usefulness against each other\n",
    "knowledgebase_query = (\n",
    "    sl.Query(\n",
    "        paragraph_index,\n",
    "        weights={\n",
    "            relevance_space: sl.Param(\"relevance_weight\"),\n",
    "            recency_space: sl.Param(\"recency_weight\"),\n",
    "            usefulness_space: sl.Param(\"usefulness_weight\"),\n",
    "        },\n",
    "    )\n",
    "    .find(paragraph)\n",
    "    .similar(relevance_space, sl.Param(\"search_query\"))\n",
    "    .select_all()\n",
    "    .limit(sl.Param(\"limit\"))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "492bba0f-a341-4052-82d8-8bf69467511b",
   "metadata": {},
   "source": [
    "## Perform retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d6e38954-73da-4c3f-9545-3dc3451c7d23",
   "metadata": {},
   "outputs": [],
   "source": [
    "initial_query_text: str = \"What should management monitor?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85ae6a7d-d4b2-418d-9861-d4300dc1492c",
   "metadata": {},
   "source": [
    "First, let's do a simple retrieval based only on text similarity. We can see that the recency-, and usefulness weight is set to 0, so the document creation dates or usefulness scores are not affecting our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a2790c0c-b290-4ac5-8b88-c3466051d719",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "only_relevance_result = app.query(\n",
    "    knowledgebase_query,\n",
    "    relevance_weight=1,\n",
    "    recency_weight=0,\n",
    "    usefulness_weight=0,\n",
    "    search_query=initial_query_text,\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b7e9a7-04e7-4c03-b602-2fb70f1acae3",
   "metadata": {},
   "source": [
    "Let's observe the difference between elements with ids 23 and 0 respectively. These documents essentially say the same thing, except that the old document (id 0) prescribes a bi-annual, while the fresh one (id 23) an annual review on policies and procedures. Also notice that an element (id 34) with very low usefulness score appears it the top 10 documents when only accounting for relevance. Let's see what happens if we upweight recency!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "704f409e-9496-4dc1-9734-33164b505731",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mild_recency_result = app.query(\n",
    "    knowledgebase_query,\n",
    "    relevance_weight=1,\n",
    "    recency_weight=0.10,\n",
    "    usefulness_weight=0,\n",
    "    search_query=initial_query_text,\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459c594b-c49a-4cdd-af37-bb1ea2221338",
   "metadata": {},
   "source": [
    "Now, note how the relative position of element 23 compared to 0 improved as it received a score boost for its recency. It is a useful feature that enables us to differentiate between similar documents based on their freshness - let's boost that and see if we can have only recent, not outdated, information in our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1b14b33a-36a3-412f-bf7c-5f88262ed84e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normal_recency_result = app.query(\n",
    "    knowledgebase_query,\n",
    "    relevance_weight=1,\n",
    "    recency_weight=0.4,\n",
    "    usefulness_weight=0,\n",
    "    search_query=initial_query_text,\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd4f0cf8-d133-4b36-923e-25554d27bcbb",
   "metadata": {},
   "source": [
    "Yes, we can! Now that all results are fresh, the outdated counterparts were removed from the result set. This way we can supply nice and clean context to the generation model.\n",
    "\n",
    "Nevertheless, we still have the usefulness issue. Observe the interesting dynamics here between documents with IDs 24 and 35 - doc 24 softened the requirement to \"wherever applicable\" from \"in all cases\". Also, doc 34 and 35 are deemed useless by the employees - looking at the text it is easy to understand why. We need to incorporate usefulness into our search, as recency is not able to differentiate between these two documents, and they are both relevant to our query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b4ceecf4-f59e-41a2-8cc3-44ab073b29a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normal_recency_usefulness_result = app.query(\n",
    "    knowledgebase_query,\n",
    "    relevance_weight=1,\n",
    "    recency_weight=0.4,\n",
    "    usefulness_weight=0.5,\n",
    "    search_query=initial_query_text,\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec7dffd9-96e9-4ee7-8e1b-ad4bdc802b14",
   "metadata": {},
   "source": [
    "Giving some emphasis to usefulness eradicated all the seemingly irrelevant docs from our results!\n",
    "\n",
    "It is also important to see that the same weights produce relevant, useful results for queries about parental leave — a topic parts of which are only discussed in the older HR document in an useful way (maternity leave) or in the fresh but otherwise useless policy document (paternal leave). We should be able to receive results about both topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "493ef42f-4d45-4bda-bb5f-af56f27569fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "maternity_result = app.query(\n",
    "    knowledgebase_query,\n",
    "    relevance_weight=1,\n",
    "    recency_weight=0.4,\n",
    "    usefulness_weight=0.5,\n",
    "    search_query=\"What are the companies terms for parental leave?\",\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d089467-225a-448d-bfa2-e7a6ba76b6e3",
   "metadata": {},
   "source": [
    "Yes, our weights work in this case too! We can go on and create our chatbot on top of the retrieval we set up."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "856a0101-48cb-41e1-9432-fe144e828ff9",
   "metadata": {},
   "source": [
    "# NEW Natural Language Querying"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea189e7-2263-4148-abf8-6e124a5adcf3",
   "metadata": {},
   "source": [
    "This way we can save the trouble of setting the weights and we can let the system set them based on the detected user intent. Check the feature out in more depth in the corresponding [feature notebook](https://github.com/superlinked/superlinked/blob/main/notebook/feature/natural_language_querying.ipynb)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "24f1602a-7538-49fa-8eb4-f250f3fdc04b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# switch to your working OpenAI API key!\n",
    "openai_config = sl.OpenAIClientConfig(api_key=os.environ[\"OPEN_AI_API_KEY\"], model=\"gpt-4o\")\n",
    "\n",
    "nlq_knowledgebase_query = knowledgebase_query.with_natural_query(sl.Param(\"natural_query\"), openai_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7cb502d0-c58a-4239-b131-f845220712f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>usefulness</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "      <th>date_created</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.</td>\n",
       "      <td>0.469966</td>\n",
       "      <td>23</td>\n",
       "      <td>0.545108</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.</td>\n",
       "      <td>0.555771</td>\n",
       "      <td>0</td>\n",
       "      <td>0.522990</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.</td>\n",
       "      <td>0.830746</td>\n",
       "      <td>24</td>\n",
       "      <td>0.507148</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.</td>\n",
       "      <td>0.410012</td>\n",
       "      <td>2</td>\n",
       "      <td>0.491230</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>1</td>\n",
       "      <td>0.489609</td>\n",
       "      <td>2022-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.</td>\n",
       "      <td>0.551468</td>\n",
       "      <td>25</td>\n",
       "      <td>0.489070</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.</td>\n",
       "      <td>0.030283</td>\n",
       "      <td>34</td>\n",
       "      <td>0.468849</td>\n",
       "      <td>2023-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.</td>\n",
       "      <td>0.665170</td>\n",
       "      <td>17</td>\n",
       "      <td>0.466823</td>\n",
       "      <td>2023-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                 body  \\\n",
       "0                                         Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.   \n",
       "1                                                    Management is tasked with ensuring that information security policies and procedures undergo an bi-annual review, are disseminated and accessible, and that employees and contractors adhere to them throughout their tenure or contract period. The yearly policy assessment must encompass a scrutiny of any interconnected or referenced protocols, standards, or directives.   \n",
       "2  Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.   \n",
       "3                                            Management must guarantee that information security obligations are conveyed to individuals, either through written job outlines, policies, or alternative documented means that are regularly updated and managed accurately. Compliance with information security protocols and the fulfillment of related duties should form part of the performance evaluation process in all cases.   \n",
       "4                                                                                                                                                                                                                                                                               Management should factor in undue pressures and potential fraud scenarios when devising incentives and delineating roles, duties, and authorizations.   \n",
       "5                                                                                                                                                                                                                                                                   Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.   \n",
       "6                                                                                                                                                                                                                                                                                  Management should factor in undue pressures and potential fraud scenarios when devising splorps and delineating roles, duties, and authorizations.   \n",
       "7                                                                                                                                                                                                                                                                                           To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.   \n",
       "\n",
       "   usefulness  id  similarity_score date_created  \n",
       "0    0.469966  23          0.545108   2023-12-31  \n",
       "1    0.555771   0          0.522990   2022-12-31  \n",
       "2    0.830746  24          0.507148   2023-12-31  \n",
       "3    0.410012   2          0.491230   2022-12-31  \n",
       "4    0.310004   1          0.489609   2022-12-31  \n",
       "5    0.551468  25          0.489070   2023-12-31  \n",
       "6    0.030283  34          0.468849   2023-12-22  \n",
       "7    0.665170  17          0.466823   2023-12-31  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlq_result = app.query(\n",
    "    nlq_knowledgebase_query,\n",
    "    natural_query=\"recent useful info on management responsibilities\",\n",
    "    limit=TOP_N,\n",
    ")\n",
    "\n",
    "df = sl.PandasConverter.to_pandas(only_relevance_result)\n",
    "sl.PandasConverter.format_date_column(df, \"created_at\", \"date_created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14cad191-6a81-4456-a0f8-7611a344c2f8",
   "metadata": {},
   "source": [
    "## Augmentation - formulate query for our LLM for generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8df1176-7a04-4680-bdf9-33d6ac5da76e",
   "metadata": {},
   "source": [
    "To keep this light and tool independent, we just manually craft our query template based on [LLama2 instructions from HuggingFace](https://huggingface.co/blog/llama2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ee35ac56-dc6f-4256-86ea-95ae1089da64",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_items_from_retrieval: int = 5\n",
    "context_text: str = (\n",
    "    \"\\n\"\n",
    "    + \"\\n\".join(\n",
    "        f\n",
    "        for f in sl.PandasConverter.to_pandas(normal_recency_usefulness_result)[\"body\"].iloc[\n",
    "            :context_items_from_retrieval\n",
    "        ]\n",
    "    )\n",
    "    + \"\\n\"\n",
    ")\n",
    "\n",
    "rag_query = f\"\"\"<s>[INST] <<SYS>>\n",
    "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.\n",
    "Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.\n",
    "Please ensure that your responses are socially unbiased and positive in nature.\n",
    "\n",
    "If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct.\n",
    "If you don't know the answer to a question, please don't share false information.\n",
    "<</SYS>>\n",
    "\n",
    "Please answer the following question by using information from the provided context information!\n",
    "CONTEXT_INFORMATION: {context_text}\n",
    "QUESTION: {initial_query_text}\n",
    "[/INST]\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7ce8d635-2e93-4f86-8042-67ef12e1dc2a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<s>[INST] <<SYS>>\n",
      "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.\n",
      "Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.\n",
      "Please ensure that your responses are socially unbiased and positive in nature.\n",
      "\n",
      "If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct.\n",
      "If you don't know the answer to a question, please don't share false information.\n",
      "<</SYS>>\n",
      "\n",
      "Please answer the following question by using information from the provided context information!\n",
      "CONTEXT_INFORMATION: \n",
      "Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.\n",
      "Information security leaders and managers shall ensure appropriate professional development occurs to provide an understanding of current threats and trends in the security landscape. Security leaders and key stakeholders shall attend trainings, obtain and maintain relevant certifications, and maintain memberships in industry groups as appropriate.\n",
      "All <Company Name> employees will undergo an annual performance review which will include an assessment of job performance, competence in the role, adherence to company policies and code of conduct, and achievement of role-specific objectives.\n",
      "To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.\n",
      "Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.\n",
      "\n",
      "QUESTION: What should management monitor?\n",
      "[/INST]\n"
     ]
    }
   ],
   "source": [
    "print(rag_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d700545e-45ec-43c7-b222-9e69e6f7d9ab",
   "metadata": {},
   "source": [
    "## Generation - prompt the LLM for text generation to answer our query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "6da4d2b1-af52-4d2b-9b73-70ec6cfe6a9b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "You are trying to access a gated repo.\nMake sure to have access to it at https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.\n401 Client Error. (Request ID: Root=1-6797aa67-1804d5c66ea816964daa063f;6b08aa50-3ac9-46ab-a5c8-59ac814a5057)\n\nCannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/resolve/main/config.json.\nAccess to model meta-llama/Llama-2-7b-chat-hf is restricted. You must have access to it and be authenticated to access it. Please log in.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py:406\u001b[0m, in \u001b[0;36mhf_raise_for_status\u001b[0;34m(response, endpoint_name)\u001b[0m\n\u001b[1;32m    405\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 406\u001b[0m     \u001b[43mresponse\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    407\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m HTTPError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/requests/models.py:1024\u001b[0m, in \u001b[0;36mResponse.raise_for_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1023\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m http_error_msg:\n\u001b[0;32m-> 1024\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(http_error_msg, response\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "\u001b[0;31mHTTPError\u001b[0m: 401 Client Error: Unauthorized for url: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/resolve/main/config.json",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mGatedRepoError\u001b[0m                            Traceback (most recent call last)",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/utils/hub.py:403\u001b[0m, in \u001b[0;36mcached_file\u001b[0;34m(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_gated_repo, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)\u001b[0m\n\u001b[1;32m    401\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    402\u001b[0m     \u001b[38;5;66;03m# Load from URL or cache if already cached\u001b[39;00m\n\u001b[0;32m--> 403\u001b[0m     resolved_file \u001b[38;5;241m=\u001b[39m \u001b[43mhf_hub_download\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    404\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpath_or_repo_id\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    405\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfilename\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    406\u001b[0m \u001b[43m        \u001b[49m\u001b[43msubfolder\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43msubfolder\u001b[49m\u001b[43m)\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01melse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43msubfolder\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    407\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrepo_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrepo_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    408\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrevision\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrevision\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    409\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcache_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcache_dir\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    410\u001b[0m \u001b[43m        \u001b[49m\u001b[43muser_agent\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43muser_agent\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    411\u001b[0m \u001b[43m        \u001b[49m\u001b[43mforce_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    412\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    413\u001b[0m \u001b[43m        \u001b[49m\u001b[43mresume_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    414\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    415\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlocal_files_only\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlocal_files_only\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    416\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    417\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m GatedRepoError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py:114\u001b[0m, in \u001b[0;36mvalidate_hf_hub_args.<locals>._inner_fn\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    112\u001b[0m     kwargs \u001b[38;5;241m=\u001b[39m smoothly_deprecate_use_auth_token(fn_name\u001b[38;5;241m=\u001b[39mfn\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, has_token\u001b[38;5;241m=\u001b[39mhas_token, kwargs\u001b[38;5;241m=\u001b[39mkwargs)\n\u001b[0;32m--> 114\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:862\u001b[0m, in \u001b[0;36mhf_hub_download\u001b[0;34m(repo_id, filename, subfolder, repo_type, revision, library_name, library_version, cache_dir, local_dir, user_agent, force_download, proxies, etag_timeout, token, local_files_only, headers, endpoint, resume_download, force_filename, local_dir_use_symlinks)\u001b[0m\n\u001b[1;32m    861\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 862\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43m_hf_hub_download_to_cache_dir\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    863\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# Destination\u001b[39;49;00m\n\u001b[1;32m    864\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcache_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcache_dir\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    865\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# File info\u001b[39;49;00m\n\u001b[1;32m    866\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrepo_id\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrepo_id\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    867\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfilename\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfilename\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    868\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrepo_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrepo_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    869\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrevision\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrevision\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    870\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# HTTP info\u001b[39;49;00m\n\u001b[1;32m    871\u001b[0m \u001b[43m        \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    872\u001b[0m \u001b[43m        \u001b[49m\u001b[43metag_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43metag_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    873\u001b[0m \u001b[43m        \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    874\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    875\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    876\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;66;43;03m# Additional options\u001b[39;49;00m\n\u001b[1;32m    877\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlocal_files_only\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlocal_files_only\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    878\u001b[0m \u001b[43m        \u001b[49m\u001b[43mforce_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    879\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:969\u001b[0m, in \u001b[0;36m_hf_hub_download_to_cache_dir\u001b[0;34m(cache_dir, repo_id, filename, repo_type, revision, endpoint, etag_timeout, headers, proxies, token, local_files_only, force_download)\u001b[0m\n\u001b[1;32m    968\u001b[0m     \u001b[38;5;66;03m# Otherwise, raise appropriate error\u001b[39;00m\n\u001b[0;32m--> 969\u001b[0m     \u001b[43m_raise_on_head_call_error\u001b[49m\u001b[43m(\u001b[49m\u001b[43mhead_call_error\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mforce_download\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlocal_files_only\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    971\u001b[0m \u001b[38;5;66;03m# From now on, etag, commit_hash, url and size are not None.\u001b[39;00m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:1484\u001b[0m, in \u001b[0;36m_raise_on_head_call_error\u001b[0;34m(head_call_error, force_download, local_files_only)\u001b[0m\n\u001b[1;32m   1482\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(head_call_error, RepositoryNotFoundError) \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(head_call_error, GatedRepoError):\n\u001b[1;32m   1483\u001b[0m     \u001b[38;5;66;03m# Repo not found or gated => let's raise the actual error\u001b[39;00m\n\u001b[0;32m-> 1484\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m head_call_error\n\u001b[1;32m   1485\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1486\u001b[0m     \u001b[38;5;66;03m# Otherwise: most likely a connection issue or Hub downtime => let's warn the user\u001b[39;00m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:1376\u001b[0m, in \u001b[0;36m_get_metadata_or_catch_error\u001b[0;34m(repo_id, filename, repo_type, revision, endpoint, proxies, etag_timeout, headers, token, local_files_only, relative_filename, storage_folder)\u001b[0m\n\u001b[1;32m   1375\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1376\u001b[0m     metadata \u001b[38;5;241m=\u001b[39m \u001b[43mget_hf_file_metadata\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1377\u001b[0m \u001b[43m        \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43metag_timeout\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\n\u001b[1;32m   1378\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1379\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m EntryNotFoundError \u001b[38;5;28;01mas\u001b[39;00m http_error:\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py:114\u001b[0m, in \u001b[0;36mvalidate_hf_hub_args.<locals>._inner_fn\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    112\u001b[0m     kwargs \u001b[38;5;241m=\u001b[39m smoothly_deprecate_use_auth_token(fn_name\u001b[38;5;241m=\u001b[39mfn\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m, has_token\u001b[38;5;241m=\u001b[39mhas_token, kwargs\u001b[38;5;241m=\u001b[39mkwargs)\n\u001b[0;32m--> 114\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:1296\u001b[0m, in \u001b[0;36mget_hf_file_metadata\u001b[0;34m(url, token, proxies, timeout, library_name, library_version, user_agent, headers)\u001b[0m\n\u001b[1;32m   1295\u001b[0m \u001b[38;5;66;03m# Retrieve metadata\u001b[39;00m\n\u001b[0;32m-> 1296\u001b[0m r \u001b[38;5;241m=\u001b[39m \u001b[43m_request_wrapper\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1297\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mHEAD\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1298\u001b[0m \u001b[43m    \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1299\u001b[0m \u001b[43m    \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1300\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallow_redirects\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m   1301\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfollow_relative_redirects\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m   1302\u001b[0m \u001b[43m    \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1303\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1304\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1305\u001b[0m hf_raise_for_status(r)\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:277\u001b[0m, in \u001b[0;36m_request_wrapper\u001b[0;34m(method, url, follow_relative_redirects, **params)\u001b[0m\n\u001b[1;32m    276\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m follow_relative_redirects:\n\u001b[0;32m--> 277\u001b[0m     response \u001b[38;5;241m=\u001b[39m \u001b[43m_request_wrapper\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    278\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    279\u001b[0m \u001b[43m        \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    280\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfollow_relative_redirects\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    281\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    282\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    284\u001b[0m     \u001b[38;5;66;03m# If redirection, we redirect only relative paths.\u001b[39;00m\n\u001b[1;32m    285\u001b[0m     \u001b[38;5;66;03m# This is useful in case of a renamed repository.\u001b[39;00m\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/file_download.py:301\u001b[0m, in \u001b[0;36m_request_wrapper\u001b[0;34m(method, url, follow_relative_redirects, **params)\u001b[0m\n\u001b[1;32m    300\u001b[0m response \u001b[38;5;241m=\u001b[39m get_session()\u001b[38;5;241m.\u001b[39mrequest(method\u001b[38;5;241m=\u001b[39mmethod, url\u001b[38;5;241m=\u001b[39murl, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mparams)\n\u001b[0;32m--> 301\u001b[0m \u001b[43mhf_raise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresponse\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    302\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m response\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py:423\u001b[0m, in \u001b[0;36mhf_raise_for_status\u001b[0;34m(response, endpoint_name)\u001b[0m\n\u001b[1;32m    420\u001b[0m     message \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m    421\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;241m.\u001b[39mstatus_code\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m Client Error.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot access gated repo for url \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;241m.\u001b[39murl\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    422\u001b[0m     )\n\u001b[0;32m--> 423\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m _format(GatedRepoError, message, response) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n\u001b[1;32m    425\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m error_message \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAccess to this resource is disabled.\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n",
      "\u001b[0;31mGatedRepoError\u001b[0m: 401 Client Error. (Request ID: Root=1-6797aa67-1804d5c66ea816964daa063f;6b08aa50-3ac9-46ab-a5c8-59ac814a5057)\n\nCannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/resolve/main/config.json.\nAccess to model meta-llama/Llama-2-7b-chat-hf is restricted. You must have access to it and be authenticated to access it. Please log in.",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[44], line 7\u001b[0m\n\u001b[1;32m      3\u001b[0m model \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmeta-llama/Llama-2-7b-chat-hf\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m      5\u001b[0m \u001b[38;5;66;03m# we use the pure transformers library without popular wrappers\u001b[39;00m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;66;03m# but LLamaIndex and LangChain can easily be plugged in here as those use transformers under the hood too\u001b[39;00m\n\u001b[0;32m----> 7\u001b[0m tokenizer \u001b[38;5;241m=\u001b[39m \u001b[43mAutoTokenizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_pretrained\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      8\u001b[0m generation_pipeline \u001b[38;5;241m=\u001b[39m pipeline(\n\u001b[1;32m      9\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext-generation\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m     10\u001b[0m     model\u001b[38;5;241m=\u001b[39mmodel,\n\u001b[1;32m     11\u001b[0m     torch_dtype\u001b[38;5;241m=\u001b[39mfloat16,\n\u001b[1;32m     12\u001b[0m     device_map\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mauto\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m     13\u001b[0m )\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py:878\u001b[0m, in \u001b[0;36mAutoTokenizer.from_pretrained\u001b[0;34m(cls, pretrained_model_name_or_path, *inputs, **kwargs)\u001b[0m\n\u001b[1;32m    876\u001b[0m         config \u001b[38;5;241m=\u001b[39m AutoConfig\u001b[38;5;241m.\u001b[39mfor_model(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mconfig_dict)\n\u001b[1;32m    877\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 878\u001b[0m         config \u001b[38;5;241m=\u001b[39m \u001b[43mAutoConfig\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_pretrained\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    879\u001b[0m \u001b[43m            \u001b[49m\u001b[43mpretrained_model_name_or_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrust_remote_code\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtrust_remote_code\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\n\u001b[1;32m    880\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    881\u001b[0m config_tokenizer_class \u001b[38;5;241m=\u001b[39m config\u001b[38;5;241m.\u001b[39mtokenizer_class\n\u001b[1;32m    882\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(config, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mauto_map\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAutoTokenizer\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m config\u001b[38;5;241m.\u001b[39mauto_map:\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py:1021\u001b[0m, in \u001b[0;36mAutoConfig.from_pretrained\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m   1018\u001b[0m trust_remote_code \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtrust_remote_code\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m   1019\u001b[0m code_revision \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcode_revision\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m-> 1021\u001b[0m config_dict, unused_kwargs \u001b[38;5;241m=\u001b[39m \u001b[43mPretrainedConfig\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_config_dict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpretrained_model_name_or_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1022\u001b[0m has_remote_code \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mauto_map\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m config_dict \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAutoConfig\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m config_dict[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mauto_map\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[1;32m   1023\u001b[0m has_local_code \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel_type\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m config_dict \u001b[38;5;129;01mand\u001b[39;00m config_dict[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel_type\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;129;01min\u001b[39;00m CONFIG_MAPPING\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/configuration_utils.py:590\u001b[0m, in \u001b[0;36mPretrainedConfig.get_config_dict\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m    588\u001b[0m original_kwargs \u001b[38;5;241m=\u001b[39m copy\u001b[38;5;241m.\u001b[39mdeepcopy(kwargs)\n\u001b[1;32m    589\u001b[0m \u001b[38;5;66;03m# Get config dict associated with the base config file\u001b[39;00m\n\u001b[0;32m--> 590\u001b[0m config_dict, kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_get_config_dict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpretrained_model_name_or_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    591\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m config_dict \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    592\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m {}, kwargs\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/configuration_utils.py:649\u001b[0m, in \u001b[0;36mPretrainedConfig._get_config_dict\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m    645\u001b[0m configuration_file \u001b[38;5;241m=\u001b[39m kwargs\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_configuration_file\u001b[39m\u001b[38;5;124m\"\u001b[39m, CONFIG_NAME) \u001b[38;5;28;01mif\u001b[39;00m gguf_file \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m gguf_file\n\u001b[1;32m    647\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    648\u001b[0m     \u001b[38;5;66;03m# Load from local folder or from cache or download from model Hub and cache\u001b[39;00m\n\u001b[0;32m--> 649\u001b[0m     resolved_config_file \u001b[38;5;241m=\u001b[39m \u001b[43mcached_file\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    650\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpretrained_model_name_or_path\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    651\u001b[0m \u001b[43m        \u001b[49m\u001b[43mconfiguration_file\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    652\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcache_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcache_dir\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    653\u001b[0m \u001b[43m        \u001b[49m\u001b[43mforce_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mforce_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    654\u001b[0m \u001b[43m        \u001b[49m\u001b[43mproxies\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproxies\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    655\u001b[0m \u001b[43m        \u001b[49m\u001b[43mresume_download\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume_download\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    656\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlocal_files_only\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlocal_files_only\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    657\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtoken\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtoken\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    658\u001b[0m \u001b[43m        \u001b[49m\u001b[43muser_agent\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43muser_agent\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    659\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrevision\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrevision\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    660\u001b[0m \u001b[43m        \u001b[49m\u001b[43msubfolder\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msubfolder\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    661\u001b[0m \u001b[43m        \u001b[49m\u001b[43m_commit_hash\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcommit_hash\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    662\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    663\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m resolved_config_file \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    664\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m, kwargs\n",
      "File \u001b[0;32m~/git/superlinked-internal/.venv/lib/python3.11/site-packages/transformers/utils/hub.py:421\u001b[0m, in \u001b[0;36mcached_file\u001b[0;34m(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_gated_repo, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m resolved_file \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m _raise_exceptions_for_gated_repo:\n\u001b[1;32m    420\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m resolved_file\n\u001b[0;32m--> 421\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mEnvironmentError\u001b[39;00m(\n\u001b[1;32m    422\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mYou are trying to access a gated repo.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mMake sure to have access to it at \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    423\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://huggingface.co/\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mpath_or_repo_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mstr\u001b[39m(e)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    424\u001b[0m     ) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n\u001b[1;32m    425\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m RepositoryNotFoundError \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    426\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mEnvironmentError\u001b[39;00m(\n\u001b[1;32m    427\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mpath_or_repo_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m is not a local folder and is not a valid model identifier \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    428\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mlisted on \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mhttps://huggingface.co/models\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mIf this is a private repository, make sure to pass a token \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    429\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhaving permission to this repo either by logging in with `huggingface-cli login` or by passing \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    430\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m`token=<your_token>`\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    431\u001b[0m     ) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n",
      "\u001b[0;31mOSError\u001b[0m: You are trying to access a gated repo.\nMake sure to have access to it at https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.\n401 Client Error. (Request ID: Root=1-6797aa67-1804d5c66ea816964daa063f;6b08aa50-3ac9-46ab-a5c8-59ac814a5057)\n\nCannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/resolve/main/config.json.\nAccess to model meta-llama/Llama-2-7b-chat-hf is restricted. You must have access to it and be authenticated to access it. Please log in."
     ]
    }
   ],
   "source": [
    "# load the 7B parameter Llama model for chat from Huggingface\n",
    "# for the first run this will download model, for subsequent ones it will load it\n",
    "model = \"meta-llama/Llama-2-7b-chat-hf\"\n",
    "\n",
    "# we use the pure transformers library without popular wrappers\n",
    "# but LLamaIndex and LangChain can easily be plugged in here as those use transformers under the hood too\n",
    "tokenizer = AutoTokenizer.from_pretrained(model)\n",
    "generation_pipeline = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model,\n",
    "    torch_dtype=float16,\n",
    "    device_map=\"auto\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e6e8adf0-098c-4269-9292-06efff7cf40e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result: <s>[INST] <<SYS>>\n",
      "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.\n",
      "Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.\n",
      "Please ensure that your responses are socially unbiased and positive in nature.\n",
      "\n",
      "If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct.\n",
      "If you don't know the answer to a question, please don't share false information.\n",
      "<</SYS>>\n",
      "\n",
      "Please answer the following question by using information from the provided context information!\n",
      "CONTEXT_INFORMATION: \n",
      "Management shall ensure that information security responsibilities are communicated to individuals, through written job descriptions, policies or some other documented method which is accurately updated and maintained. Compliance with information security policies and procedures and fulfillment of information security responsibilities shall be evaluated as part of the performance review process wherever applicable.\n",
      "Management shall be responsible for ensuring that information security policies and procedures are reviewed annually, distributed and available, and that employees and contractors abide by those policies and procedures for the duration of their employment or engagement. Annual policy review shall include a review of any linked or referenced procedures, standards or guidelines.\n",
      "Information security leaders and managers shall ensure appropriate professional development occurs to provide an understanding of current threats and trends in the security landscape. Security leaders and key stakeholders shall attend trainings, obtain and maintain relevant certifications, and maintain memberships in industry groups as appropriate.\n",
      "To ensure that employees and contractors meet security requirements, understand their responsibilities, and are suitable for their roles.\n",
      "Management shall consider excessive pressures, and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.\n",
      "\n",
      "QUESTION: What should management monitor?\n",
      "[/INST]  Based on the provided context information, management should monitor:\n",
      "\n",
      "1. Information security responsibilities of individuals, which includes ensuring that job descriptions, policies, and procedures are accurately updated and maintained.\n",
      "2. Compliance with information security policies and procedures as part of the performance review process.\n",
      "3. Review of information security policies and procedures annually, including linked or referenced procedures, standards, or guidelines.\n",
      "4. Professional development of information security leaders and managers to stay current with threats and trends in the security landscape.\n",
      "5. Suitability of employees and contractors for their roles, including monitoring for excessive pressures and opportunities for fraud when establishing incentives and segregating roles, responsibilities, and authorities.\n",
      "\n",
      "It is important to note that management should monitor these areas in a socially unbiased and positive manner, ensuring that any monitoring is conducted in a fair and ethical manner, without any harmful or illegal content.\n"
     ]
    }
   ],
   "source": [
    "# we prompt the LLM for text generation with our RAG query\n",
    "sequences = generation_pipeline(\n",
    "    rag_query,\n",
    "    do_sample=True,\n",
    "    top_k=10,\n",
    "    num_return_sequences=1,\n",
    "    eos_token_id=tokenizer.eos_token_id,\n",
    "    pad_token_id=tokenizer.eos_token_id,\n",
    ")\n",
    "for seq in sequences:\n",
    "    print(f\"Result: {seq['generated_text']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba6a539-144c-46f5-bee8-88e64f34ae87",
   "metadata": {},
   "source": [
    "## Evaluation of the answer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aaf7e68-3847-4336-a420-e429ee470743",
   "metadata": {},
   "source": [
    "The results are structured, contain the information present in the context documents, and are tailored to our question. Based on hardware, it will run in a manageable time. \n",
    "In 3), we can see that the generated text contains the correct annual term. This is mainly because it was fed the right thing as context information."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
